The UMHexagonS motion estimation algorithm in H.264 was studied, and an improved fast motion estimation algorithm was proposed. First, the fixed search range, the unsymmetrical cross search, the 5×5 small rectangular spiral search, the uneven multi-hexagon-grid search and the extended hexagon-based search were analyzed. Then the optimized search modes were given respectively, which called dynamic search window, adaptive rood pattern search, the directional 3×3 small rectangular search pattern, the predictive intensive direction search and the modified extended hexagon-based search. Thus Adaptive Pattern Direction Search (APDS) algorithm was formed by these optimized search modes. The experimental results conducted on different test sequences show that, compared to UMHexagonS algorithm, the APDS algorithm can save about 29.64% Motion Estimation (ME) time and reduce the average number of checking points per Motion Vector (MV) generation about 21.64, while incurring nothing obvious loss in the reconstructed picture quality and less increment in the bit rate. With the efficiency improvement of ME, the real-time performance of the encoder is further enhanced.